The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated (UTC)
docs	Ready	Preview, Comment	Jul 8, 2026 12:13am

PR Summary

Medium Risk
Changes the deploy-critical migration runner and performs a user_table_rows data backfill during upgrade; design is idempotent with per-table locking and retry-on-failure, but a failed deploy blocks until the backfill succeeds.

Overview
Adds run-once TypeScript data migrations to the existing db:migrate flow so self-hosted deploys get the same backfills as hosted, with no compose/helm changes.

After Drizzle SQL migrations succeed (still under the migration advisory lock), runScriptMigrations runs pending entries from an append-only registry, records each name in a runner-managed script_migrations table, and clears statement_timeout / lock_timeout on that session so long backfills can wait on app locks instead of failing with 55P03. assertLockSessionHeld is extracted and invoked before the script phase so a recycled connection cannot run scripts without the lock.

The order_key backfill moves from deleted apps/sim/scripts/backfill-table-order-keys.ts into registry entry 0001_backfill_table_order_keys: same per-table advisory lock and position-ordered keys, with chunked updates via unnest (two bind params per chunk) instead of large VALUES lists. Per-table errors are logged and the migration throws so the journal row is not written and only still-unkeyed tables retry.

fractional-indexing is published as @sim/utils/fractional-indexing (tests included) so the slim migrations image can use it; order-key.ts keeps a thin wrapper over that package. The standalone repair script’s docs now point at the script migration for NULL backfills vs collation repair.

^{Reviewed by Cursor Bugbot for commit 4983f86. Bugbot is set up for automated code reviews on this repo. Configure here.}

@greptile review

Greptile Summary

This PR adds a run-once TypeScript script migration framework to the existing db:migrate entrypoint, ensuring self-hosted deployments automatically run TS data migrations (previously manual) with zero config changes. It ports the order_key fractional-indexing backfill as 0001, moves fractional-indexing.ts to @sim/utils for availability in the slim migrations Docker image, and deletes the now-redundant standalone backfill script.

• New packages/db/script-migrations/: ScriptMigration interface with authoring rules, an explicit registry, and a runner (runScriptMigrations) invoked from migrate.ts after SQL migrations succeed within the same advisory-lock session; applied scripts are journaled in a runner-managed script_migrations table.
• 0001_backfill_table_order_keys: Ports the order_key backfill with per-table advisory locks, chunked unnest-based UPDATEs instead of VALUES lists, per-table error isolation, and idempotent retry semantics.
• assertLockSessionHeld(): Extracted from runMigrationsWithRetry into a shared helper called both at each DDL retry and once before script migrations run, closing a session-recycle window.

Confidence Score: 5/5

Safe to merge — the migration framework is additive and well-guarded; the backfill is idempotent and isolated per table; the only finding is a detached JSDoc block from the assertLockSessionHeld refactor.

The core machinery (advisory-lock guard, timeout resets, ON CONFLICT DO NOTHING journal insert, per-table error isolation) is all correct and addresses issues raised in prior review rounds. The backfill logic is idempotent: re-runs skip fully keyed tables, the unnest-based UPDATE avoids parameter limits, and per-table failures let the migration stay pending and retry only unfinished tables. The only finding is a cosmetic orphaned comment left behind when assertLockSessionHeld was extracted from runMigrationsWithRetry.

packages/db/scripts/migrate.ts has a minor doc inconsistency from the refactor; all other files are clean.

Important Files Changed

_{Reviews (4): Last reviewed commit: "improvement(db): re-verify advisory-lock..." | Re-trigger Greptile}

@greptile review

Greptile Summary

This PR introduces a run-once TypeScript script-migration mechanism layered on top of the existing db:migrate entrypoint, so self-hosted deployments automatically apply one-off data backfills without any additional orchestration. It also relocates the dependency-free fractional-indexing implementation into @sim/utils so the slim migrations image can import it.

• packages/db/script-migrations/ — new ScriptMigration interface, explicit append-only registry, and a runner that creates the script_migrations tracking table, deduplicates names, checks required env vars, runs pending migrations in order, and records each by name only after up() resolves (mirroring drizzle's __drizzle_migrations journal).
• 0001_backfill_table_order_keys — ports the old standalone backfill script as the first registry entry; uses per-table advisory locks and unnest-based chunked UPDATEs; per-table failures are isolated so remaining tables still run, then the migration throws to stay pending for retry.
• packages/utils/src/fractional-indexing.ts — 100%-identical rename from apps/sim/lib/fractional-indexing/; exported via the new ./fractional-indexing entry in packages/utils/package.json.

Confidence Score: 4/5

The migration framework and backfill implementation are well-structured — advisory locking, per-table isolation, idempotency, and retry semantics are all correct. Two small gaps in migrate.ts are safe in every primary scenario but could surface as operational friction on live self-hosted databases.

The core design is sound and end-to-end tested. The lock_timeout = '5s' left over from the DDL phase is inherited by backfill DML statements; for the primary target (self-hosted, app down during migration) there is no contention and no impact, but a live upgrade with concurrent row writes could see sporadic per-table failures that require an extra upgrade cycle to clear. The missing pid/session check before runScriptMigrations is very unlikely to matter given max_lifetime: null, but it breaks the symmetry of the existing guard in runMigrationsWithRetry. Neither issue introduces data loss or corruption.

packages/db/scripts/migrate.ts — the two gaps described above both live here. All other files look correct.

Important Files Changed

Sequence Diagram

%%{init: {'theme': 'neutral'}}%%
sequenceDiagram
    participant Runner as migrate.ts
    participant PG as PostgreSQL
    participant SM as script_migrations table

    Runner->>PG: connectWithRetry()
    Runner->>PG: pg_try_advisory_lock(MIGRATION_LOCK_KEY) [session-level]
    Runner->>PG: "SET statement_timeout=0, lock_timeout='5s'"
    Runner->>PG: drizzle migrate() — SQL migrations
    PG-->>Runner: SQL migrations applied
    Note over Runner: runScriptMigrations(client)
    Runner->>PG: CREATE TABLE IF NOT EXISTS script_migrations
    Runner->>SM: SELECT name FROM script_migrations
    SM-->>Runner: applied set
    loop For each pending script migration
        Runner->>Runner: check requiredEnv
        Runner->>PG: BEGIN (sql.begin)
        Runner->>PG: pg_advisory_xact_lock(table-key) [tx-level]
        Runner->>PG: "SELECT id FROM user_table_rows WHERE table_id=X ORDER BY position"
        Runner->>Runner: generateNKeysBetween(null, null, n)
        loop chunks of 5000
            Runner->>PG: UPDATE user_table_rows SET order_key FROM unnest(ids, keys)
        end
        Runner->>PG: COMMIT
        Runner->>SM: INSERT INTO script_migrations (name)
    end
    Runner->>PG: pg_advisory_unlock(MIGRATION_LOCK_KEY)
    Runner->>PG: warnOnInvalidIndexes()

%%{init: {'theme': 'base', 'themeVariables': {"darkMode": true, "background": "#0d1117", "primaryColor": "#21262d", "primaryTextColor": "#e6edf3", "primaryBorderColor": "#8b949e", "lineColor": "#8b949e", "textColor": "#e6edf3", "edgeLabelBackground": "#161b22", "actorBkg": "#21262d", "actorBorder": "#8b949e", "actorTextColor": "#e6edf3", "actorLineColor": "#8b949e", "signalColor": "#8b949e", "signalTextColor": "#e6edf3", "noteBkgColor": "#373320", "noteBorderColor": "#d4a72c", "noteTextColor": "#f0e6c0", "labelBoxBkgColor": "#21262d", "labelBoxBorderColor": "#8b949e", "labelTextColor": "#e6edf3", "loopTextColor": "#e6edf3", "activationBkgColor": "#30363d", "activationBorderColor": "#8b949e"}}}%%
sequenceDiagram
    participant Runner as migrate.ts
    participant PG as PostgreSQL
    participant SM as script_migrations table

    Runner->>PG: connectWithRetry()
    Runner->>PG: pg_try_advisory_lock(MIGRATION_LOCK_KEY) [session-level]
    Runner->>PG: "SET statement_timeout=0, lock_timeout='5s'"
    Runner->>PG: drizzle migrate() — SQL migrations
    PG-->>Runner: SQL migrations applied
    Note over Runner: runScriptMigrations(client)
    Runner->>PG: CREATE TABLE IF NOT EXISTS script_migrations
    Runner->>SM: SELECT name FROM script_migrations
    SM-->>Runner: applied set
    loop For each pending script migration
        Runner->>Runner: check requiredEnv
        Runner->>PG: BEGIN (sql.begin)
        Runner->>PG: pg_advisory_xact_lock(table-key) [tx-level]
        Runner->>PG: "SELECT id FROM user_table_rows WHERE table_id=X ORDER BY position"
        Runner->>Runner: generateNKeysBetween(null, null, n)
        loop chunks of 5000
            Runner->>PG: UPDATE user_table_rows SET order_key FROM unnest(ids, keys)
        end
        Runner->>PG: COMMIT
        Runner->>SM: INSERT INTO script_migrations (name)
    end
    Runner->>PG: pg_advisory_unlock(MIGRATION_LOCK_KEY)
    Runner->>PG: warnOnInvalidIndexes()

_{Reviews (2): Last reviewed commit: "fix(db): reset session lock_timeout befo..." | Re-trigger Greptile}

@greptile review